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EXECUTIVE SUMMARY 


P arents and policymakers have long sought to measure the quality of their public schools and to 
report that publicly in ways that are fair and equitable. In recent years, with a renewed focus on 
student outcomes, this effort has become a very public and sometimes acrimonious debate. 

With this project, ECS sought to answer three key questions from various stakeholders in a way that 
assists parents and policymakers in creating school accountability systems or “report cards” that are 
transparent and effective. 

The key questions we asked: 

^ Of researchers - Are the report cards easy to find? 

^ Of parents - Are the report cards easy to understand? 

^ Of experts - What indicators are essential for measuring school and district performance? 

The responses, in brief: 

Researchers agreed upon eight state report cards as easy-to-find, informative and readable. Their top 
three picks are in bold: 

-r Arizona Delaware ^ Massachusetts 

^ Illinois ^ Kentucky Maine 

^ Ohio Louisiana 

Parents identified six state report cards as the best of the 50 states, based on ease of reading, providing 
sufficient data and overall usefulness. Their top three picks are in bold: 

^ Delaware ^ Arkansas 

^ District of Columbia Ohio 

Illinois Wisconsin 

Experts selected five indicators they see as essential for any state’s school accountability system: 

^ Student achievement 
^ Student academic growth 
^ Achievement gap closure 
^ Graduation rates 
^ Postsecondary and career readiness 

The co-authors of this report then reviewed ECS’ 50-state accountability database , released in January, 
and identified 13 states that are both including all five essential indicators in calculating their state school 
reports and publicly reporting all five indicators. Those 13 states: 



California 

Y New Mexico 


Tennessee 


Colorado 

^ North Carolina 


Utah 


Florida 

Ohio (final element coming in 2015) 


Wisconsin 


Kentucky 

^ Oklahoma 




Louisiana 

Pennsylvania 




Interestingly, different states excelled in different aspects considered in this project. At ECS, we believe 
states can improve their education systems by learning from each other. We hope this report assists in 
those continuing efforts. 
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INTRODUCTION 


S tate leaders are striving to increase 

transparency about how well their public 
schools are educating children. The result is 
an increase in the information about schools’ 
challenges and successes being shared with their 
communities through annual reports, often in the 
form of “report cards.” This wave of accountability 
makes it important — now more than ever — to 
analyze which measures best signal the quality 
of schools and how that information is effectively 
shared and used to improve performance. 

Transparency is important but, unlike in years 
past, it is not itself the end goal. Ultimately, today’s 
accountability systems are designed to hold schools 
responsible for their contribution to students’ 
postsecondary success and to equip parents with 
the information they need to insist upon change 
if they don’t believe their children are being well- 
served. Valid metrics are necessary if policymakers 
are to implement meaningful school ranking 
systems and, subsequently, school improvement 
plans that parents and others can trust. 


This report includes input from three different 
groups in an attempt to help state policymakers 
create accessible, useful and effective school report 
cards. 

The key questions and responding groups: 

1. Are the report cards easy to find? 
Experienced researchers at the Education 
Commission of the States (ECS) were asked 
to find selected state report cards online to 
determine the accessibility of the cards. 

2. Are they understandable to parents? 
More than a dozen parents were asked 
to rate the report cards on a 1-5 scale in 
the categories of “easy to read,” “provides 
sufficient data” and “useful.” 

3. What are best practices? 

Finally, a dozen experts convened to discuss 
the essential metrics for any accountability 
system, key considerations for policymakers 
and important decision points. 


ACCOUNTABILITY EFFORTS; A NATIONAL EVOLUTION 


State school accountability systems, and their goals, have evolved over the years: 

^ Accountability 1.0 (1900-80) - Accreditation: Initially based on inputs such as staff degrees 
and numbers of library books, this version evolves in the 1980s into a focus on performance. 

^ Accountability 2.0 (1990-2001) - Standards-Based Accountability: State lawmakers set 
academic standards and begin state testing, sometimes with rewards and/or sanctions. 

Florida launches the first state school report cards, grading schools from A to F. 

Accountability 3.0 (2001-10) - No Child Left Behind: Federal lawmakers mandate state testing 
and outline incentives and consequences with an unprecedented level of detail. Parents in 
some states receive report cards with two sets of ratings, state and federal. 

^ Accountability 4.0 (2010-present) - Race to the Top: With the renewal of NCLB stalled in 
Congress, President Obama entices states to implement reforms, such as linking student test 
scores to teacher evaluations, with Race to the Top grants. 

^ Accountability 5.0 (2013-present) - Standards, Round 2: States adopting standards such as 
the Common Core are figuring out new assessments and tweaking accountability systems to 
measure and report results. 
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Door plates to D’s: Common indicators of today’s report cards 


States have long sought to publicly report school 
quality but the measures used to determine quality 
look much different today than they did 100 years 
ago. As early as 1897, the state of Minnesota 
enacted a law requiring schools to meet certain 
minimum requirements to receive state aid. In 
1907, Illinois began awarding door plates to schools 
it deemed “superior.” And by 1925, 30 state 
departments of education were publicly reporting 
on factors such as the number of teachers with 
academic and professional qualifications and the 
frequency of community meetings.^ 

Today, every state annually publishes individual 
district and school report cards to provide a 
snapshot of how well that district and school is 
educating its students. The metrics used vary but 
the focus has clearly shifted from inputs, such 
as the number of library books in a school, to 
outcomes, such as student academic growth on 
state exams. Door plates have given way to report 
card rating systems including A-F grades, 1 to 5 
stars, numerical index scores, colors such as green 


for good schools and red for struggling schools, 
or various descriptors, such as a “continuous 
improvement” or “reward” school. 

Researchers at the Education Commission of the 
States compiled a 50-state database of what’s 
measured and reported by each state. What’s 
measured and what’s reported are not necessarily 
identical. States may measure various data and 
use that information in calculating a final letter 
grade, index score, color or descriptor. But not 
all data collected by all states is factored into 
such calculations: some states simply report out 
additional information for the public to see. 

As part of this report, ECS convened a School 
Accountability Advisory Group to discuss which 
measures should be included in every state’s 
accountability system. The members, listed in the 
appendix, identified five essential indications. The 
indicators, and the states currently measuring and 
reporting those indicators according to the ECS 
accountability database , are shown below. 


STATES AND THE FIVE ESSENTIAL INDICATORS FOR SCHOOL ACCOUNTABILITY 

Data from ECS’ 50-state database on school accountability systems show which states are using the indicators: 


Indicator Used for 
School Accountability 

No. of States Measuring 

No. of States Reporting 

Student achievement 

50 + Washington, D.C. 

50 + D.C. 

Student academic growth 

42 + D.C. 

34 + D.C. 

Achievement gap closure 

36 + D.C. 

39 + D.C. 

Graduation rates 

50 + D.C. 

50 + D.C. 

Postsecondary and 
career readiness 

19 (explicit mention; 22 if 
count proxies for readiness) 

13 (27 if count 
proxies for readiness) 


Source: Education Commission of the States, http://www.ecs.orq/htmI/educationissues/accountabiIity/stacc intro.asp. ^ 


What’s the difference between what’s measured and what’s reported? 

What’s measured refers to data that states use in calculating their school performance ratings. What’s reported 
refers to data that states make publicly available but do not necessarily include in those calculations. Twenty-two 
states include all five essential indicators in measuring school performance: Alabama (2015-16), Alaska, California, 
Colorado, Florida, Hawaii, Indiana, Kentucky, Louisiana, Minnesota, New Mexico, North Carolina, Ohio, Oklahoma, 
Pennsylvania, South Carolina, Tennessee, Texas, Utah, Virginia, Wisconsin and Wyoming. 

What is meant by postsecondary and career readiness indicators or their proxies? 

Some states explicitly refer in their accountability laws to postsecondary and career readiness indicators while 
others use indicators that serve to suggest such readiness, including college-going rates and ACT/SAT results. 
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Communication and trust: Two factors that matter, but aren’t rated 

ECS’ review of school accountability systems found calculations used by states to reach a school’s final 
grade or rating are rarely simple, often relying on algebraic equations and other mathematical formulas. 
While this may be necessary to ensure numerous indicators are represented and to create the most 
accurate ratings, such formulas can be difficult to communicate clearly to the public. 

Teachers, parents and communities like to have a basic understanding about how a school’s grade was 
derived. Weights and proportions matter. States can measure carefully selected indicators of quality but 
if the indicators are weighted incorrectly — at least, according to some observers — the result can be a 
grade or rating that some members of the public see as inaccurate and, worse, intentionally so. 

Trust is an issue. This is not surprising since the results of school ratings can range from accolades to 
staff firings to closures. Letter grades are easiest for parents and other constituents to understand. But if 
a clear rating sits atop a hill of measures that communities don’t trust, questions are likely to follow. 

Where does it go wrong? Here are some common complaints: 

^ The metrics aren’t right. For example, too much emphasis is placed on test performance and/or 
too few subjects are tested. 

^ The metrics, weights, measures and formula do not accurately reflect school performance. 

^ Composite scores are seen as less transparent and nuanced than separate indicators. 

^ Communication about how the grades are determined is vague or inconsistent. 

Even a rocket scientist can’t figure out the formula. 

^ The metrics, weights, formula and report card do not reflect public values. 

Creating a robust, valid and easy-to-understand report card is harder than it sounds. State legislatures 
and departments of education have worked years to create such report cards — only to be rewarded with 
a cacophony of criticism from their constituents. The rest of this paper is divided into three sections — 
researchers, parents and experts — that seek to help state policymakers get it right. 


IT'S COMPLICATED: 

ATTEMPTING TO OVERCOME "COMPOSITION BIAS" 


An issue with nearly every performance indicator is composition bias. Simply stated, this refers to 
the correlation between a school’s student demographics and its performance levels. Attempts to 
resolve this concern have resulted in greater attention to academic growth, rather than absolute 
performance levels, and a number of more complicated accountability systems. 

For example, states may use regression analysis, a statistical process for estimating the relationships 
among variables, to determine the weight to give poverty. Or a state may use value-added modeling, 
charting student progress over time, in an attempt to determine teaching contributions to student 
growth. While these techniques may be used to improve accuracy, they can be difficult to easily 
explain in communications about accountability systems. 
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WHAT’S THE SECRET FORMULA? IT HAS TO BE UNDERSTANDABLE! 


Examples of easy-to-understand state report card formulas include Louisiana, 
one of the top states selected by researchers and experts. 


HOW ARE SCHOOL GRADES CALCULATED? 

storting with the 2012-2013 school year, the Louisiana Department of Edocotion hos improved 
the way schools ore groded by aligning with higher standards, rewording the gams schools have 
already mode, and focusing on students below grode level. This means; 
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Source: http://www.louisianaschools.net/docs/test-results/8-19-13-report-card-infoqraphic.pdf?sfvrsn=6 
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SECTION I: RESEARCHERS 


Are the report cards easy to find? 

Researchers with the Education Commission of 
the States were assigned to find state report cards 
online in an effort to see how easy the cards are to 
locate. They were given the name of a particular 
school in a particular state and asked to find its 
most recent report card. One goal was to ascertain 
the level of computer skill required to find the state- 
issued cards. In many cases, private school-rating 
websites such as GreatSchools.org , city-data.com 
or 50Can.org came up first in computer searches, 
while serious diligence and technical understanding 
were needed to find the state-sponsored reports. 

The three researchers were asked to rate each 
report card from 1 (unsatisfactory) to 3 (excellent) 


in the following categories: Findable, Readable, 
Understandable and Graphics. For the latter 
category, the question was “Were graphics used 
well to convey the information?” Even those 
experienced in online research had difficulty: “I 
wasn’t able to find school-level report cards,” 
lamented one while another noted, “Could not find 
using a Google search - lots of confounding search 
results.” They identified eight report cards as 
above average in all categories: Arizona , Delaware , 
Illinois , Kentucky , Louisiana , Massachusetts, 

Maine and Ohio . Of those, they agreed Arizona, 
Illinois and Ohio had overall the best easy-to-find, 
informative and readable report cards. 


RESEARCHERS' RATINGS: "THESE STATES DO IT BEST! " 

ARIZONA 

Summary: This report card received excellent ratings in nearly all categories. It was particularly noted 
for being easy to find and to understand, though the PDF version of the card was not rated as highly. 

“The simple format is very reader-friendly. All the essential information is present 
and easy to process ... The graphics are well-done and convey information at a glance.” 
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ILLINOIS 

Summary: Given top marks in most categories, this report card was particularly noted for being easy to 
understand and for its use of graphics. Also praised: Links allowing readers to “drill down” to learn more. 

“I really like the overview on the first page with the snapshot and basic graphs. It made the basic 
information very easy to understand and to digest. I also liked how the graphics were interactive. ” 
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Summary: Another report card with nearly perfect scores, Ohio’s effort was lauded for its graphics and 
for being easy to read and understand. One concern: Several data points are labeled “Coming in 2015.” 
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RESEARCHER REVIEW TIKES" 


IS THE REPORT CARD EASY TO FIND? 

“It was relatively easy to find (after minimal digging) and 1 like that you can download the report” 

“The school-specific information did not come up through an Internet search, 
but found relatively easily through the state education department" 

IS THE REPORT CARD EASY TO READ? 

“The report card was very good. Easy to read. Not too much information shown, but links to more detailed 

information were easily accessible.” 

“1 also liked that information was available in Spanish.” 

IS THE REPORT CARD EASY TO UNDERSTAND? 

“1 like that there’s a two-page snapshot as well as the more detailed online version. 
Information was broken down into tabs, which 1 think is helpful.” 

“Nice balance of data and narrative explanation. Tor Parents’ and ‘for Educators’ are GREAT features to see.” 

DOES THE USE OF GRAPHICS HELP CONVEY INFORMATION? 

“The graph titles also provide additional information by hovering over the text. ” 

“1 really like the overview on the first page with the snapshot and basic graphs. 

It made the basic information very easy to understand and digest. 

1 also liked how the graphics were interactive and allow users to click through for more details.” 


RESEARCHER REVIEW "DISLIKES" 

IS THE REPORT CARD EASY TO FIND? 

“When 1 searched for report cards on the Department of Education site, the first link it brought up was broken. 
It took me nine minutes to get to the accountability reporting system page.” 

“Found right away with a Google search, but the website doesn’t work right with Firefox. Worked fine with MS Explorer.” 

IS THE REPORT CARD EASY TO READ? 

“This report card was clearly not designed with parents in mind. It looks like it’s just to meet state/or federal 
reporting requirements. There’s no explanation of the contents and no total score or rating.” 

“1 don’t think the format (requires lots of clicks) is user-friendly.” 

IS THE REPORT CARD EASY TO UNDERSTAND? 

“Oddly, the school’s A- F grade doesn’t appear on the report. You have to go to the Excel spreadsheet to get the A-F 
grade. There’s information on the web page about how the grades are calculated, but you have to be willing 

to click and read several different documents.” 

“1 see that the school got a four-star rating, but 1 don’t see any content around that. Four out of what? Five? Ten?” 

DOES THE USE OF GRAPHICS HELP CONVEY INFORMATION? 

“There are a bunch of nice charts and graphs, but you have to click on each thing separately to see them. ” 

“Nearly unreadable. It was very difficult to understand what was being tracked or scored.” 
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SECTION II: PARENTS 


Do the report cards contain useful 

To determine how useful the report cards actually 
are to parents, ECS asked parents from across 
the country to follow a link to an individual school 
report card from each of the 50 states. The 14 
parents were selected by ECS staff and represent 
a mix of educational attainment, ethnicity, income 
levels and geography, both in terms of urban/rural 
and in terms of U.S. states. Their children range in 
age from kindergarten to high school. 

Each of the parents reviewed report cards from all 
50 states and rated them from 1 (unacceptable) 
to 5 (excellent) in the categories of “easy to 
read,” “provides sufficient data” and “useful.” ECS 
selected for review a mix of elementary, middle 
and high schools that were moderately diverse in 
student population and that received ratings in the 
moderate to upper range.^ 


information? 

Overall, the parents favored report cards with clear 
graphics that made the data easy to understand. 
They also liked when additional information was 
available if a viewer wanted to drill down. However, 
there was not always consistent agreement. On 
the same high school report card, for example, one 
parent labeled the report card as unacceptable in 
each category while another parent labeled the 
report card as excellent in all categories. 

Report cards from Illinois and the District of 
Columbia were identified as favorites by a majority 
of parents, or eight of the 14. They were closely 
followed by Delaware (chosen by six parents) and 
then Arkansas , Ohio and Wisconsin (each selected 
by five parents). 


PARENTS SPEAK: "THESE STATES GOT IT RIGHT!" 

DISTRICT OF COLUMBIA 

Summary: Parents raved about the “very clear” presentation of information and features such as the 
ability to compare schools and the option to ask for more data via a readily available email form. 

“Wow!! This is one of my favorites. The ability to ‘explore’ the data is really nice. No other school we 
looked at had this feature,” said one parent while another noted, “I wanted to read it more.” 
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ILLINOIS 

Summary: Parents applauded this site for being easy for navigate, noting its clear directions and 
‘appealing’ graphics. They liked the ability to compare schools and to convert information to Excel. 

“Fabulous graphics on Fast Facts front page. Also, terrific tech use of ‘scan QR code’ on the At-A- 
Glance report,” said one parent while another noted “The whole website is really easy to interpret. 
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DELAWARE 

Summary: Parents were enthusiastic about the inclusion of more staff data than other states and the 
ability to drill down from tabs labeled School, Student and Staff. A common refrain: “User friendly.” 

“Loved this one - especially the school, teachers, students tabs to help sort out data!” said one parent 
while another commented, “Nice front-page summary, easy to drill down for more data.” 
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A Clear Winner: Illinois 

Illinois was the only state whose school report cards, found 
easily at: www.illinoisreportcard.com , were selected in the 
top three by both researchers and parents. 

The interactive site is rich with graphics, pop-up explanations 
and links to at-a-glance reports, videos and additional 
resources. Indicators are typically accompanied by tabs 
labeled “Explanation of Display,” “Context” and “Resources.” 

An example is the display regarding student academic 
growth, a concept that can be tough to explain. Illinois uses 
a short video to explain the concept, describes how growth 
fits into the overall performance picture and links to a 
Frequently Asked Questions document prepared by the state. 

Additional comments from parents: 

“Easily accessible. ” 

“Easy to navigate. ” 

“Provided directions as to how to navigate the 
page and was not overwhelming with data. ” 

“Had links to compare the school to district & state.” 

“Very informative. ” 

Additional comments from researchers: 

“Very good. Easy to read. Not too much information 
shown, but links to more detailed information were 
easily accessible. ” 

“THE BEST SO FAR. Easy to interpret, everything is 
clickable for more information. ” 
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PARENT REVIEW TIKES" 


IS THE REPORT CARD EASY TO READ? 

“/ like that the data is presented in both table and bar graph format 
Four-color bar graph easy to decipher at a glance. ” 

“Everything is on one page. You can get additional information from just one click on the graph and 
the breakdown of data pops up. The information is very clearly presented.” 

“Tabs across top make navigation quick. ” 

DOES THE REPORT CARD PROVIDE SUFFICIENT DATA? 

“As a parent 1 could find information that would be important to me when making decisions about schools, 
t felt like 1 got an understanding of the school without going there from what is on this site.” 

“1 could learn about more than just data about the schools from this site.” 

“Very thorough - WOW! Could be a bit much to some but I’m sure most parents 
would love more information than less. ” 

IS THE REPORT CARD USEFUL? 

“Additional information such as school safety, graduation rates, etc., help to paint a whole picture of this school.” 

“Great summary/comparison to the state - demonstrating this school outperforms state average.” 

“Postsecondary and workforce readiness category is nice to know.” 


PARENT REVIEW "DISLIKES" 

IS THE REPORT CARD EASY TO READ? 

“This report made the user have to use dropdown boxes and select what you wanted to see. 
Not easy to compare everything like charts and spreadsheets/graphs. ” 

“They use words that are not meaningful to the general public (Cell Count, etc.). ” 

“+/- 1 really liked this report card although it is not supported for tablet or smartphone. ” 

DOES THE REPORT CARD PROVIDE SUFFICIENT DATA? 

“So much emphasis on enrollment in the past 10 years, but not much information on performance or assessment.” 

“Not much reference or explanation of the ‘B’ grade in the upper right-hand corner. Amount of data insufficient.” 

“No growth comparisons from years past. Data is very limited.” 

IS THE REPORT CARD USEFUL? 

“Extremely boring and data in tables not clearly labeled or explained. ” 

“Nice summary, but very little info. Would not be good if you were moving to area and wanted more school info. 

Where is the rest of the data?” 

“Like reading a corporate financial report of 20 pages to get information. 
Lot of data that is scattered and not formatted to be easily understood. ” 
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An important consideration 

Overall, parent reactions to the report cards broke down into a fairly even distribution — a third of the 
cards rose to the top, a third sank to the bottom and a third landed somewhere in the middle. Individual 
reactions to some state’s accountability reports, however, were widely disparate. A sampling of those 
opinions is presented here to further illustrate how difficult it can be to create public reporting systems 
that please everyone: 


ONE CARD, DIFFERENT RESPONSES: A MATTER OF PREFERENCE 

While many of the scores reported by the parent panel were in the same range, there were definite 
differences of opinion. 

ALASKA 

VERMONT 

PRO - “Performance index was easy to read and 
provides a good feel for each school’s 
performance” and “Good data, easy to read!” 

CON - “One 96-page document with one page for 
each school in Alaska. Rates three subjects 
and just gives percent proficient, not levels 
or what percentages were in previous 
years. No demographic or teacher data 
included. ... What is a good score?” 

PRO - “You have a lot of control in 

building the type of reports you 
want to view. If you know exactly 
what you are looking for, this is a 
useful website.” 

CON - “Vague, would like to see a grade 
in the district - A, B, C.” 
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SECTION III: EXPERTS 


Essential metrics states should use to measure school success 


Because of the complexities involved with selecting 
school measures that accurately and reliably signal 
the quality and health of schools, ECS convened a 
panel of 12 experts in December 2013 to look at 
what states measure and what they should report 
regarding the quality and health of their schools/ 
The robust discussion covered the maturation 
of state accountability and report card efforts, 
and the pitfalls facing states when the measures 
become political liabilities. The experts pinpointed 
essential metrics, caveats, key considerations and 
important policymaker decision points. 

The ECS School Accountability Advisory Group 
grappled with many questions, including: 

^ Is more information necessarily better? 

^ Do metrics and formulas accurately 
measure which schools are doing well? 

-f What level of data is necessary? 
Student-level or cohort-level? 

Is there an absolute level on an indicator 
below which no school should operate? 

^ Do you weigh progress toward a goal or 
an absolute measure? 

^ Since you cannot account for everything, 
what are the best metrics for examining 
the health of a school or system? 

^ How do you ensure growth toward a goal 
is recognized while not losing focus on 
reaching the goal? 

Key Findings: 

1. Set a clear goal or “North Star” 

The expert group noted that states need a clear 
goal or “North Star” of what they are trying to 
accomplish with renewed school improvement 
efforts. 

For example, Kentucky lists its “College or career 
ready for all” goal with their formula and on the 
state landing page for its school report cards. 


Or, if a state such as Massachusetts wants to 
focus on a P-20 system, measures should signal 
success throughout that system. That might 
mean inclusion of a pre-K indicator. Creating a 
common goal for the state encourages public 
buy-in and a cohesive message. 

When choosing the indicators or metrics to 
measure school performance, experts say it is 
important to link the causes, interventions and 
reliable outcomes that will lead to achieving the 
overall goal or “North Star.” 

2. Beware unintended consequences 

Prior to delving into essential indicators for 
states, the experts’ panel discussed over-arching 
concerns about accountability. A major theme 
was that states and districts must be careful 
in how they hold schools accountable and how 
the information is reported to the public. That’s 
because what is measured and reported has the 
possibility of driving bad behaviors. 

For example, grading a school based on the 
number of expulsions may have the unintended 
consequence of encouraging teachers and 
administrators to be more lenient on behavioral 
infractions. 

3. Ensure state systems can handle the data 

Because the most accurate accountability 
systems typically require a reliable student-level 
data system, the experts noted policymakers 
must consider the capacity of their state 
longitudinal data system and staff when choosing 
metrics. Many state data systems were initially 
created to track school-level accountability 
data and weren’t designed to capture student- 
level data in a secure and shareable manner. 
Portability of data across schools, districts and 
platforms is critical for understanding the growth 
students are making, but existing state data 
systems may not be up to the task. 
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Five essential indicators every state should measure and report 

While the experts encouraged additional metrics based on individual state and district issues, they 
recommended every state report card include these indicators: 

Student achievement 
^ Student academic growth 
^ Achievement gap closure 
^ Graduation rates 

College and career readiness 

For each indicator, the experts examined the various metrics used, advantages, caveats and key state 
decision points. Detailed findings for each indicator are listed on the following pages.^ 
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Making the Grade: States Meeting the Five Essential Indicators 


The experts convened by ECS did not focus on how to find state report cards or, once found, how to 
navigate them. Their charge was different: Identify the essential metrics for any accountability system. 

So it may not be surprising that there is little cross-over between the top states picked by parents and 
researchers and those states identified as measuring and reporting on the five essential indicators. 

The 13 states identified as meeting the experts’ criteria are California , Colorado, Florida , Kentucky , 
Louisiana , New Mexico , North Dakota , Ohio , Oklahoma , Pennsylvania , Tennessee , Utah and Wisconsin . 

This example of a New Mexico state report card for Albuquerque High School illustrates the use of the 
five essential indicators: 



School Grade Report Card 
2013 


Certified 


ESSENTIAL 

INDICATORS 


Albuquerque High 

District: Albuquerque Public Schools 

GrtdeRjnte: 09-12 Code: 1S90 



This School 
SUUwide C B«n<hm«rk I 



Current Standing 

How did students pedonn In the most recent school year? Students 
are tested on how well they met targets for thefr grade level. 


School Possible 
Grade Paints Points 

B 18.07 30 

School Growth 

In tht past 3 y«ars did tht school incroast fradt kvitl porformanct? 
for example did this year's 10th graders improve over last year's 10th 
graden? 


B 7.30 10 

student Growth of Highest Performing Students 
How well did the school help Individual students Improve? The 
Mghest performing students arc those whose prior scores placed 
them In the lop three quarters (7SK) of their school. Indtvidual 
student growth over the past 3 years Is compared to the state 
benchmark. 


A 9.98 10 

Studtnt Growth of Lowdst Ptrforming Students 
How wdl did th« school hdp ir>drvidual students improve? The lowest 
parforming students art those whose prior scores placed them In the 
bottom quarter (2S%) of thee school. Indrvidual student growth over 
the past 3 years is compared to the state benchmark. 


A 9.56 10 

Opportunity to learn 

Does the school foster an environment that faciittates learning? Are 
teachers using recognized Instructional methods, and do students want 
to come to school? 


B 6.46 8 

Graduation 

How does the school contribute to on-time graduation? On-time mearu 
within 4 ytars. and to a lesser extent« within S and 6 years for students 
who require longer. 


C 11.96 17 

College and Career Readiness 

Are students prepared for what lies ahead alter high school? Schools 
receive credit when students participate in college entrance exams, arid 
cotirsework leading to dual credit and vocational ccrttfkaUon. The school 
receives additional credit when students meet success goals. 


A 11.92 15 

Bonus Points 

Docs the school show cxcapuooal aptitude for involving students and 
parents in education, reducing truancy, and promoting extracurricular 
activrties? 

r 

2.34 5 



3 Year 
Average 

72.7 


2011 


2012 


2013 


B 


HmI School Grade 

75.0 to 100.0 A 

65.0 to 74.9 B 

50.0 to 64.9 C 

35.0 to 49.9 D 

0.0 to 34.9 F 


Total 

Points 

77.57 


Source: http://webapp2.ped.state.nm.us/SchooiData/docs/1213/SchoolGradinq/001 590 ALBUQUERQUE PUBLIC 
SCHOOLS ALBUQUERQUE HIGH SchoolGradinq 2013.pdf 
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Essential Indicator #1: Student Achievement 


Every state gives students standards-based assessments and reports those results to schools and 
parents. States choose the subjects to be tested and set the cut scores necessary for students to show 
proficiency. Reporting overall or absolute levels of student achievement typically indicates the number or 
percentage of a school’s students who are deemed to be performing proficiently in particular subjects. 
Many states have defined proficient as achieving grade-level expectations. 

But many students come to schools with significant disadvantages. Some states, such as Tennessee, 
seek to accommodate for such disadvantages with statistical models. These models attempt to reduce 
the likelihood that schools serving large numbers of disadvantaged students will have their performance 
designation affected by conditions over which they have little control. 

Including absolute levels of student achievement as an indicator in an accountability system is typically 
seen as an advantage for schools serving more affluent populations. To balance that concern, many 
states include changes in school achievement levels over time in their ratings formulas and some include 
student academic growth measures. In addition, a number of states have created comparisons among 
schools of similar demographics. California, for example, ranks its schools statewide and compares each 
school to another 100 schools with similar rates of poverty, parent education and other indicators. 


FACTORS FOR 

POLICYMAKERS TO CONSIDER: 


^ Critics believe a focus on test scores may 
create a “high-stakes” environment for 
students, teachers and administrators. 

^ Communities may have a hard time rallying 
behind the tests without alignment between 
the tests, grade levels and learning 
requirements. 

^ Setting the cut scores for proficiency on the 
tests is not a perfect science. 

^ If tests change, school accountability 

systems should too. When moving to a new 
assessment, states should carefully align 
the old and new tests to validate that the 
standards are being met. 


QUESTIONS FOR 
POLICYMAKERS TO CONSIDER: 


^ Which subjects will be tested and in which 
grades? 

Do the tests fully align to the standards and 
do they meet college- and career-ready 
expectations? 

How are the cut scores for the assessments 
determined? Who makes those decisions and 
how often will the cut scores be re-examined? 

^ Will the results for groups of students, such as 
English language learners, minorities or low- 
income students, be explicitly reported as part 
of the accountability system? Will these results 
factor in a school’s final ranking or grade? 

Does the accountability system consider trend 
data, such as the past two or three years, or is 
it based on one year’s results? 

^ Will end-of-course exams or other 

assessments, such as college entrance tests 
including the ACT or SAT, be included in the 
school and district rating system? 
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Essential Indicator #2: Student Academic Growth 


A small but increasing number of states are refining their accountability systems to measure and 
reward student academic growth. Based on a review of students’ test score gains from previous grades, 
researchers can predict the amount of growth those students are likely to make in a given year and then 
compare to actual performance. This differs from changes in school-level performance over time because 
actual individual student performance is tracked, even as students move in and out of schools. 

This prediction can help determine whether a student is making expected progress in a particular subject. 
Measuring student academic growth is one way of analyzing test data to measure teaching and learning. 
It’s often referred to as “value-added” or looking to see whether a teacher has added value to a student’s 
body of knowledge. 

In addition, measuring student academic growth and using past growth to predict future results can 
be used as part of “catch up” or “keep up” indicators. The “catch up” indicator examines the progress 
of lower-performing students who need to catch up to the performance of their peers. The “keep up” 
indicator looks at the growth of the highest-performing students, who may stagnate if growth isn’t 
recognized as a priority. 

Measuring and reporting student academic growth is generally seen as a way of resolving concerns about 
composition bias and of recognizing schools and districts that are working hard, even if their results fall 
short of absolute performance goals. 


FACTORS FOR 

POLICYMAKERS TO CONSIDER: 


^ “Growth” is often perceived as being too confusing 
— people may not understand it because the 
underlying statistical calculations are complex and 
not easily replicated by non-statisticians. 

^ Communication strategies for explaining growth 
are critically important. It is possible to keep the 
explanations simple, even if the methodology is 
complex. 

^ Because simple growth models depend largely 
on the formula determining individual student 
growth, it is possible to game the system and 
make the data look better than it actually is. 
Calculations should address students who switch 
schools midyear, those who start or finish a 
course outside of the normal academic calendar, 
who have missing data or those who are far below 
or above grade level for their cohort. 

^ Attempting to control for student demographics 
may increase the precision of results in models 
that don’t use all available prior achievement data, 
but it might have the effect of implying there are 
different standards for different students. 


QUESTIONS FOR 
POLICYMAKERS TO CONSIDER: 


Will growth be measured against 
an absolute proficiency standard or 
against “peer” schools with similar 
demographics? 

^ How can growth calculations keep from 
working against or accommodate for 
high-performing schools with less room 
for growth? Does your state rating 
formula ensure that achievement 
growth within the highest-performing 
quartile also matters? 

^ Will student academic growth be 
considered in evaluating teacher 
performance? If so, does the system 
used for determining growth align with 
what’s needed to measure teacher 
performance? 
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Essential Indicator #3: Achievement Gap Closure 


Gaps in achievement separating groups of students by income and ethnicity have been the focus of 
numerous studies, policy innovation and public concern for many years. Researchers have identified 
a variety of factors that appear related to these achievement gaps, including family income, parent 
education levels, access to high-quality preschool, peer influences, curricular and instructional quality, 
and teacher expectations. 

Many states have chosen to focus on these particular achievement gaps as a means of ensuring 
progress — or a lack thereof — is highlighted. Equally as important, however, are indicators that focus 
on achievement gaps such as those between English language learners and native English speakers, 
students performing in the lowest quartile versus those performing better, male students and female 
students, and so on. In short, the intent of reporting and/or measuring achievement gaps should be to 
ensure that all students are being served. 

It’s also important to consider the size of the groupings used in this analysis. For example, the 
performance of all boys versus all girls in a school may not be useful. But a further breakdown by 
academic subject and grade may yield more helpful data. 


FACTORS FOR 

POLICYMAKERS TO CONSIDER: 


^ While challenging, experts agree it is 

important to measure and report disparities in 
performance levels among different groups of 
students. 

^ Closing achievement gaps should benefit all 
students - accelerating the growth of lower 
performers without reducing growth in higher 
achievers. 

^ In addition to subgroups based on student 
demographics, consider subgroups based 
solely on achievement. For example, closing 
gaps between historically struggling and 
higher-performing readers in a grade level or 
school. 

^ Decisions surrounding determination of 
subgroup size matter. Subgroup size can 
enhance fairness but the use of “super 
subgroups” — such as grouping all ethnicities 
under the term minority versus breakdowns 
by individual ethnicity — may risk covering up 
low performance by smaller subgroups. 

^ Federal regulations governing the reporting 
of assessment results for minimum sample 
sizes, to avoid releasing personally identifiable 
information, should be consulted. 


QUESTIONS FOR 
POLICYMAKERS TO CONSIDER: 


^ Which achievement measures will be used 
— test scores, graduation rates, growth, 
etc.? 

4- Which subgroups should be included 
and which excluded — by income, race, 
achievement level, etc.? 

Are achievement gaps measured within 
schools and within districts? 

^ Are multiple years of data used for school 
performance measures? 

^ Should performance measures specifically 
target academic growth of the lowest 
quartile by giving that group additional 
weight in the accountability formula? 

^ Flow can unintended consequences of 
subgroup size be accommodated in small, 
rural schools? 
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Essential Indicator #4: Graduation Rates 


Measuring graduation rates is intended to encourage all schools to ensure all students complete 
requirements to receive a diploma. The credential, which data has long demonstrated results in better 
employment prospects and higher pay, can have a profound impact on student life outcomes. 

The U.S. Department of Education’s required calculation for a school’s four-year graduation rate is to 
divide the number of students graduating in four years with a regular high school diploma by the number 
of students who entered the school as freshmen four years previously. This calculation is adjusted to 
account for student movement in and out of the school during the four-year period. 

A graduation rate would seem to be a fairly easy metric on its face. Yet it offers a myriad of complexities 
when considering how to encourage schools to serve students who might “count” against them, such 
as those who have left school and returned or who have been slow to accumulate enough credits to 
graduate. For example, how does a state consider students who take five or six years to graduate? Such 
decisions can have a significant influence on the effort schools put forth in educating at-risk students. 


FACTORS FOR 

POLICYMAKERS TO CONSIDER: 


^ Allowing credit for five-year and six-year 
graduation rates, in addition to the four- 
year rate, could encourage schools to work 
with struggling students. 

^ Alternately, does allowing credit for five- 
year and six-year graduation rates reduce 
pressure to help students reach credential 
completion within four years? 

"F Because graduation requirements differ in 
states, with some requiring end-of-course 
exams versus credit accumulation, accurate 
cross-state comparisons are difficult. 

"F Managing student mobility data requires a 
strong longitudinal data tracking system. 

"F Even with common calculations, schools 
have the potential to “game the system” 
by being selective about which students 
are included in a four-year graduation 
rate. 


QUESTIONS FOR 
POLICYMAKERS TO CONSIDER: 


"F Should five-year and six-year graduation 
rates be included in the state accountability 
system to encourage schools to work with 
struggling students? 

"F Will a school’s graduation rate be measured 
against an absolute goal, such as 100 percent, 
or a state average when determining a grade 
or score for the report card? 

"F Similarly, should a school’s graduation rate be 
compared against demographically similar or 
“peer” schools, all schools or perhaps both? 

"F Will trend data, such as three years’ worth 
of graduation rates, be used to determine if 
progress is being made? 

"F Consider potential loopholes schools might 
use to improve their ratings, such as excluding 
some students, and figure out how to close 
them. 

"F Is there a minimum graduation rate below 
which a school would fall into the lowest 
performance category? 
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Essential Indicator #5: Postsecondary and Career Readiness 


While many states are working to define postsecondary and career readiness, the ECS School 
Accountability Advisory Panel defined it as when a student can perform college level-work without the 
need for remediation. Often, the more explicit definition in terms of metrics is provided at the state level. 
An indicator of career readiness creates the need for clarity in defining what career-ready looks like. 

These indicators of postsecondary and career readiness were commonly used by states: 


^ Dual enrollment participation and/or completion 

> Advanced Placement participation and/or results 

ACT/SAT participation and/or results 

^ International Baccalaureate program 
participation 

^ College-going rate 


^ Percentage of students taking algebra in grade 8 

Industry certifications earned 

Percentage of students enrolled in 
postsecondary programs 

^ Percentage of students assessed as needing 
college remediation 


FACTORS FOR 

POLICYMAKERS TO CONSIDER: 


^ No single formula or definition guarantees freshman- 
year college success. 

^ States must increase the dialogue between all aspects 
of K-12 and postsecondary education to create an 
aligned P-20 system. Each part of the system provides 
a necessary building-block for postsecondary success or 
workforce readiness. Those blocks must be aligned for 
individual college- and career-readiness measures. 

^ Measures related to dual enrollment should recognize 
that dual enrollment may be limited by student location 
or availability of online courses. Additionally, whether 
students take part in dual enrollment may be limited by 
counseling availability and teacher support. 

^ When including courses and tests that students select 
into, such as Advanced Placement, ACT and SAT, 
include both the course or test-taking and the course 
or test-passing rates. 

^ Including Advanced Placement participation and results 
in an accountability formula bring into question the 
availability of courses offered in person and online and 
test cut scores. 

^ Determining whether students entered college ready 
to perform college-level work requires a relatively 
stable student population and a strong longitudinal data 
tracking system. 


QUESTIONS FOR 
POLICYMAKERS TO CONSIDER: 


^ What other metrics might 
be considered to measure 
postsecondary or career readiness? 
Is the data capability available to 
measure those? 

^ Which advanced offerings, such as 
Advanced Placement, International 
Baccalaureate or dual enrollment 
courses, are available to all 
students? 

Does the state have the 
longitudinal student-level data 
necessary to determine if students 
are successful in postsecondary 
education and/or the workforce? 

^ Do the state metrics accurately tell 
the story of whether K-12 students 
are attending college without the 
need for remediation? 
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CONCLUSION 


F or more than a century, states have created 
different ways of reporting on the quality of 
their public schools. It’s only in the last 30 
years, however, that the reporting has shifted 
from inputs to outcomes and to how well children 
are being served. This is a dramatic change and 
one that likely will continue to evolve. 

Increased public reporting about school 
performance has prompted concerns about the 
fairness of comparing schools serving different 
populations. Many states have sought to address 
this issue by compensating for poverty, which is 
linked to many out-of-school factors affecting 
achievement, in some way in their district and 
school rating systems. Often, this has sparked 
criticism that expectations are lower for different 
groups of students. Balancing fairness for all 
schools and rigor for all students is widely viewed 
as a challenge in creating accountability systems. 

The findings of the ECS School Accountability 
Advisory Group, the results from researchers 
and the survey of parents make it clear that 
communication of a state’s overarching goal 
for schools is imperative. To what end are 
schools being graded? Schools have long served, 
and continue to serve, as community centers. 
Accountability systems impacting schools carry 
the potential for disrupting communities. For 


a state school and district rating system to be 
most effective, students, parents, teachers, 
administrators, policymakers, employers and 
community members must understand the state’s 
goal and what their schools are doing — or not 
doing — to achieve it. 

Is your state’s “North Star” ensuring college 
and career readiness for all? Is it graduating 
students with 21st century skills? Is it serving 
the whole child? Is it reducing the gap between 
high-achieving and low-achieving students and 
providing opportunities for all students? Is it 
providing an accurate picture of school quality — 
or the lack thereof? 

As states continue with their efforts, some may 
need to re-evaluate their ratings systems and 
make necessary course corrections to reach their 
goals. State leaders should consider whether 
the public reports are providing increased 
transparency and serving the needs of parents 
and communities. A perfect metric, accountability 
formula or school report card does not exist. 
There is always room for improvement and the 
accountability landscape will continue to evolve. 
The key is to determine which metrics will drive 
the desired outcomes and whether measuring, 
reporting, incentivizing or leveling sanctions will 
best move the state closer to its goal. 
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APPENDIX 


Members of the ECS School Accountability Advisory Group 

The Education Commission of the States convened its School Accountability Advisory Group on 
Dec. 12-13, 2013 in Denver. Members are the following: 

^ Facilitator - Christopher Cross 

Chairman of Cross & Joftus, LLC and an ECS 2014 Distinguished Senior Fellow 
^ Jean-Claude Brizard 

President, UpSpring Education and former Chief Executive Officer, Chicago Public Schools 
^ Sandy Kress 

Partner, Akin, Gump, Straus, Hauer & Feld, LLP 
^ Eric Lerum 

Vice President for National Policy, Students First 
^ Patricia Levesque 

Chief Executive Officer, Foundation for Excellence in Education 
^ Aaron Pallas 

Professor of Sociology and Education, Teachers College Columbia University 
^ Paul Reville 

Professor of Educational Policy and Administration, Harvard Graduate School of Education 
"F Joan Sullivan 

Chief Executive Officer, Partnership for Los Angeles Schools 
"F Philip “Uri” Treisman 

Executive Director, Charles A. Dana Center at the University of Texas, Austin 

"F John White 

SAS EVAAS, SAS Institute 

^ Priscilla Wohlstetter 

Senior Research Fellow, Consortium for Policy Research in Education 
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ENDNOTES 


1. Education in the States: Nationwide Development since 1990, Jim B and Edgar Fuller (editors), Pearson (Author). National 
Education Association (Publisher), 1969. 

2. Data notes for this graph: 

-f Determinations were based on statutory requirements, although we also reviewed state-requested waivers to the 
No Child Left Behind Act. Reconciling the two made it difficult to maintain accurate counts. 

+ Achievement gap elements reflect state statutory language explicitly targeting closing achievement gaps or explicit 
targeting of the lowest-performing quartile or English Language Learners. 

+ Some states explicitly measure college and/or career readiness (and measure via proxies such as ACT/SAT scores, 
dual enrollment, college-going rate, industry certifications) while others might simply measure and/or report on the 
proxies of readiness. 

3. Education Commission of the States’ School Accountability Parent Panel reviewed state school report cards between 
Jan. 20 and Feb. 10, 2014. For parent feedback, ECS selected a mix of elementary, middle and high schools that were 
moderately diverse in student population and received ratings that were in the moderate to upper range. This resulted in a 
total of 700 report card reviews - 14 parents, each reviewing 50 state school report cards = 700 report card reviews. 

4. The ECS School Accountability Advisory Group met Dec. 12-13, 2013 in Denver. Members of the group are identified by 
name and title in an appendix to this report. The group was facilitated by Christopher Cross, chairman of Cross & Joftus, 
LLC, and an ECS 2014 Distinguished Senior Fellow. 

5. Gillian Locke, Joe Ableidinger, Bryan C. Flassel and Sharon Kebschull Barrett, Virtual Schools: Assessing Progress and 
Accountability, A Final Report of Study Findings (Washington D.C.: National Charter School Resource Center at American 
Institutes for Research, February 2014), 

http://www.charterschoolcenter.orq/sites/default/files/\/irtual%20Schools%20Accountability%20Report.pdf. 


Below are links where you can find school accountability reports for each state. 


Alabama 

Idaho 

Minnesota 

North Dakota 

Vermont 

Alaska 

Illinois 

Mississippi 

Ohio 

Virginia 

Arizona 

Indiana 

Missouri 

Oklahoma 

Washington 

Arkansas 

Iowa 

Montana 

Oregon 

West Virginia 

California 

Kansas 

Nebraska 

Pennsylvania 

Wisconsin 

Colorado 

Kentucky 

Nevada 

Rhode Island 

Wyoming 

Connecticut 

Louisiana 

New Hampshire 

South Carolina 

Washington D.C. 

Delaware 

Maine 

New Jersey 

South Dakota 

American Samoa 

Florida 

Maryland 

New Mexico 

Tennessee 

Guam 

Georgia 

Massachusetts 

New York 

Texas 

Puerto Rico (Spanish) 

Hawaii 

Michigan 

North Carolina 

Utah 

U.S. Virgin Islands 
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Education Commission 
of the States 


Education Commission of the States 
700 Broadway, Suite 810 
Denver, CO 80203 
www.ecs.org 
ecs@ecs.org 
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